Label test:e2e applied for df3f0bc. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@TaylorMutch In the process of asking my coding agent questions to wrap my head around some of the implementation details here, I stumbled on the following potential issue where a silent failure can occur in the VM token rotation:

In crates/openshell-server/src/compute/mod.rs:887, the refresh sweep treats write_sandbox_token() success as a completed refresh, then optionally pushes the token over the supervisor stream. But send_sandbox_token_update() returns Ok(false) when no supervisor session is connected (crates/openshell-server/src/supervisor_session.rs:210), and the caller ignores that boolean. For VM, the driver-side write only updates the persisted sandbox request (crates/openshell-driver-vm/src/driver.rs:589); the running guest updates /opt/openshell/auth/sandbox.jwt only when it receives SandboxTokenUpdate (crates/openshell-sandbox/src/supervisor_session.rs:422). If a running VM’s supervisor stream is temporarily disconnected during refresh, the sweep records success while the guest keeps the old token. Once that old JWT expires, reconnects and outbound RPCs can keep failing because no connected session exists to receive the new token. Fix by treating Ok(false) as an undelivered refresh for supervisor-push runtimes and retrying, or by sending the latest minted token during VM supervisor session registration.

@TaylorMutch In the process of asking my coding agent questions to wrap my head around some of the implementation details here, I stumbled on the following potential issue where a silent failure can occur in the VM token rotation:

In crates/openshell-server/src/compute/mod.rs:887 (https://github.com/NVIDIA/OpenShell/pull/1639/files), the refresh sweep treats write_sandbox_token() success as a completed refresh, then optionally pushes the token over the supervisor stream. But send_sandbox_token_update() returns Ok(false) when no supervisor session is connected (crates/openshell-server/src/supervisor_session.rs:210), and the caller ignores that boolean. For VM, the driver-side write only updates the persisted sandbox request (crates/openshell-driver-vm/src/driver.rs:589); the running guest updates /opt/openshell/auth/sandbox.jwt only when it receives SandboxTokenUpdate (crates/openshell-sandbox/src/supervisor_session.rs:422). If a running VM’s supervisor stream is temporarily disconnected during refresh, the sweep records success while the guest keeps the old token. Once that old JWT expires, reconnects and outbound RPCs can keep failing because no connected session exists to receive the new token. Fix by treating Ok(false) as an undelivered refresh for supervisor-push runtimes and retrying, or by sending the latest minted token during VM supervisor session registration.

Yup, this is exactly where I've gotten hung up on this PR and haven't made much progress since I first posted it. The VM implementation in particular has been tricky, and I am definitely open to ideas here.

@TaylorMutch I think the main question I have is, is the expectation that the microvm ComputeDriver be a single-host single-player ComputeDriver option similar to podman and docker?

If so, we should be able to make certain assumptions that can make this easier to reason with. Effectively my pitch would be to toss the entire gateway managed file approach and instead use krun_add_virtiofs3 to expose the host OS directory to the VM as a read-only virtio-fs device. Then the refresh logic could be the same as it is for docker and podman. The is similar in approach to how podman --runtime krun volume bind mounts work, though I think what we would need here could be more simple and narrow in scope since we don't need to deal with the OCI or filesystem pre-prep.